Simulation?based study design accuracy weights are not generalisable and can still lead to biased meta?analytic inference: Comments on Christie et al. (2019)

Variable study quality is a challenge for all the empirical sciences, but perhaps particularly disciplines such as ecology where experimentation frequently hampered by system complexity, scale and resourcing. The resulting heterogeneity, necessity of subsequently combining results different designs, fundamental issue evidence synthesis. We welcome recognition this Christie et al. (2019) their attempt to provide generic approach assessment meta-analytic weighting through an extensive simulation study. However, we have reservations about true generality usefulness derived ‘accuracy weights’. First, simulations rely on single effect size calculation, in odd conclusion that before-after control-impact (BACI) designs are superior randomised controlled trials (RCTs), which normally considered gold standard causal inference. Second, so-called ‘study quality’ scores long been criticised epidemiological literature failing accurately summarise individual, study-specific drivers bias shown be likely retain increase variance relative meta-regression approaches explicitly model drivers. Synthesis applications. suggest ecological meta-analysts spend more time critically, transparently, appraising actual studies before synthesis, rather than relying weights or formulas solve assumed issues; sensitivity analyses hierarchical key tools work.